Skip to content

6.5. Platform Gateway

How is agentgateway deployed?

The base pins agentgateway v1.3.1 by image digest, runs it as UID/GID 65532 with a read-only root and dropped capabilities, mounts the selected config read-only, and exposes one ClusterIP service:

agentgateway:3000   MCP
agentgateway:3001   A2A
agentgateway:4000   OpenAI-compatible model
agentgateway:15020  internal metrics

Readiness/liveness probes check the MCP listener. Resource bounds keep the lab schedulable on a small node.

How is the profile selected?

The base does not hard-code a gateway ConfigMap. Each overlay adds one generated ConfigMap:

  • Local includes agentgateway/k3d and uses Ollama at host.k3d.internal:11434.
  • GKE includes agentgateway/gke and uses Vertex with backendAuth.gcp.

The Python agent still points at agentgateway:3000 and agentgateway:4000; provider movement is a data-plane change.

How does network policy reduce reachability?

Ingress rules admit only declared callers:

  • The agentops namespace can reach all gateway listeners; kagent can reach only MCP :3000 and model :4000, not A2A or metrics.
  • The raw MCP server accepts only gateway pods.
  • MLflow accepts only the OTel collector.
  • OTel accepts signals from agentops and kagent namespaces.

Egress is denied by default for every pod in the namespace, then reopened one declared flow at a time in network-policies.yaml:

spec:
  podSelector: {}
  policyTypes: [Egress]

A namespace-wide policy allows DNS to kube-system. The gateway reaches the raw MCP server (:8000), the agent A2A port (:8080), and the collector (:4317). The agent reaches only gateway :3000/:4000 and collector :4318. The collector reaches MLflow :5000, Loki :3100, and gateway metrics :15020. The raw MCP server, MLflow, and Loki get no base egress beyond DNS.

The model upstream differs per environment, so each overlay appends its own exception to agentgateway-egress. The local overlay allows TCP 11434 to any address because host.k3d.internal resolves to the Docker bridge gateway on the host, outside every pod or namespace selector. The GKE overlay allows HTTPS (Google APIs publish no stable CIDR) plus 169.254.169.252/32 on TCP 987 and 988 for Workload Identity Federation, and adds the same pair for MLflow's GCS artifacts. Those endpoint details match GKE's standard NetworkPolicy/Calico path; Dataplane V2 uses a different endpoint and is not enabled by this course's OpenTofu module.

These policies depend on a network-policy-capable cluster: k3s enforces them out of the box, and the OpenTofu module enables Calico on GKE. They do not cover the kagent control-plane namespace or cross-cluster security.

Why should an agent pod have egress rules?

Prompt injection turns a tool-using agent into a confused deputy: hostile text inside an incident, log line, or runbook can instruct the model to send whatever is in its context somewhere else. Chapter 4.6 hardens the application layer; default-deny-egress blocks undeclared destinations when those layers fail. A hijacked agent pod can still send data over its allowed model, tool, telemetry, and DNS routes, but it cannot open a direct connection to an arbitrary attacker-controlled host.

The same reasoning keeps the model-upstream exception on the gateway pod only, scoped to one port, instead of a namespace-wide allow that every compromised workload would inherit.

What do the ResourceQuota and LimitRange protect against?

The shared local cluster hosts other projects, so resource-quota.yaml caps what the agentops namespace can claim:

hard:
  requests.cpu: "2"
  requests.memory: 4Gi
  limits.cpu: "8"
  limits.memory: 9Gi
  pods: "12"

The numbers are the declared per-container requests/limits of the deployed workloads plus one surge pod per rolling deployment, so Skaffold rollouts never deadlock against the quota. A runaway or duplicated workload is rejected at admission instead of starving neighbor namespaces. Because a compute quota rejects pods without explicit resources, the LimitRange supplies bounded defaults so ad-hoc diagnostic pods remain schedulable. Inspect live consumption with:

kubectl -n agentops describe resourcequota agentops-compute

How do you debug a NetworkPolicy lockout?

Egress policies are easy to over-tighten, so practice a deliberate lockout. First verify the guardrail works: the raw MCP server has no egress allowance beyond DNS, so a connection from it to MLflow must time out even though the collector keeps delivering traces to that same endpoint:

kubectl -n agentops exec deploy/agentops-mcp -- python -c \
  'import urllib.request; urllib.request.urlopen("http://mlflow:5000/health", timeout=5)'

Expect TimeoutError — the policy, not the service, refused the flow. Now cause a real lockout and diagnose it:

kubectl -n agentops delete networkpolicy dns-egress
kubectl -n agentops exec deploy/agentops-mcp -- python -c \
  'import socket; print(socket.gethostbyname("agentgateway"))'

Name resolution fails for every pod in the namespace. Diagnose in this order:

  1. List which policies select the failing pod: kubectl -n agentops get networkpolicy and kubectl -n agentops describe networkpolicy default-deny-egress.
  2. Separate resolution from connection: retry the socket.gethostbyname probe, then the urllib probe against an allowed destination.
  3. Check that one selected policy allows the destination's port and peer; default-deny plus no matching allow is a lockout.

Restore the declared state by reapplying only the rendered policies (from infra/, so the overlay's Ollama exception is preserved):

kubectl kustomize k8s/overlays/local | yq 'select(.kind == "NetworkPolicy")' | kubectl apply -f -

How do you access the private services?

No Ingress or LoadBalancer is created. Use temporary local forwards:

kubectl -n agentops port-forward svc/agentgateway \
  3000:3000 3001:3001 4000:4000 15020:15020
kubectl -n agentops port-forward svc/mlflow 5000:5000

Use http://localhost:3001 for A2A clients. Direct agent port 8080 is reserved for diagnosis. The model listener enforces the demo API key from the agentgateway-client Secret, so a forwarded call to :4000 needs -H "Authorization: Bearer agentgateway" (Chapter 5.5).

How does GKE authenticate to Vertex?

OpenTofu creates a Google service account with roles/aiplatform.user and roles/serviceusage.serviceUsageConsumer, binds agentops/agentgateway through Workload Identity Federation, and the overlay annotates that Kubernetes service account. GKE's metadata server obtains the Kubernetes service-account assertion and exchanges it for short-lived ambient credentials; the pod does not need an automounted Kubernetes API token, and no JSON key is created or mounted.

MLflow receives a separate GSA and only roles/storage.objectUser on its artifact bucket, with the same no-API-token posture.

How do you keep secrets in git safely?

The course platform holds no real credential today: the agentgateway-client Secret carries the non-secret agentgateway marker, and the GKE path uses Workload Identity, so no key exists to store. But every real platform eventually holds one — a hosted-model API key, a Grafana admin password — and the answer to "how do I commit configuration without committing credentials" is to commit ciphertext. SOPS (MPL-2.0) with age (BSD-3-Clause) is the minimal OSS pattern: encrypted values live in git, the decryption key stays local.

The repository ships a worked example:

  1. .sops.yaml declares one creation rule: files under infra/**/secrets/ are encrypted to an age recipient, and encrypted_regex: ^(data|stringData)$ keeps kind, metadata, and comments reviewable in diffs — only the secret values become ciphertext.
  2. agentgateway-client.enc.yaml is a committed encrypted Secret: the hosted-model variant of the gateway client key, with a fake demo value as plaintext.
  3. secrets.sh wraps the lifecycle: keygen writes the age key to the gitignored infra/secrets/age.agekey, then encrypt, decrypt, and edit operate on manifests through the creation rule.

The committed recipient is a course demo key whose private half is not published, so the example ciphertext is a reference you can read, not decrypt. Make the pattern yours in three commands:

infra/scripts/secrets.sh keygen                  # prints your public recipient
# put that recipient in .sops.yaml, write your Secret manifest, then:
infra/scripts/secrets.sh encrypt infra/k8s/base/secrets/my-secret.enc.yaml
infra/scripts/secrets.sh decrypt infra/k8s/base/secrets/my-secret.enc.yaml

Be honest about the scope: this is a lab pattern, not a production KMS. Whoever holds the age private key holds every secret encrypted to it, there is no access audit, and rotation means generating a new key, re-encrypting every file, and re-issuing every credential the old key protected. Production platforms back SOPS with a cloud KMS or use an external secrets operator; the git-holds-ciphertext discipline taught here transfers unchanged.

How do you decrypt secrets at deploy time?

With one explicit, reviewable step — decrypt to stdout and pipe into kubectl, never onto disk:

infra/scripts/secrets.sh decrypt \
  infra/k8s/base/secrets/agentgateway-client.enc.yaml | kubectl apply -f -

The encrypted manifest is deliberately not listed in kustomization.yaml. Plain kustomize build cannot decrypt SOPS files, so wiring it in would either break scripts/check-infra.sh for anyone without the private key or require the ksops exec plugin and its --enable-alpha-plugins flag on every render. The zero-friction path keeps the plaintext non-secret marker in agentgateway-client.yaml inside the overlay, and the encrypted variant is applied on top only when you actually hold a hosted-model credential — the pattern lesson without a decryption dependency in the deploy loop.

sops finds the key through SOPS_AGE_KEY_FILE; the script defaults it to infra/secrets/age.agekey so decryption works with no extra setup. The GKE path needs none of this for cloud access: Workload Identity issues short-lived ambient tokens, so there is no static key to encrypt in the first place. Prefer identity federation over stored secrets wherever the provider offers it; SOPS is for the credentials that remain.

What must never be committed?

  1. The age private key. infra/secrets/, **/*.agekey, and any decrypted **/*.dec.yaml output are gitignored; verify with git check-ignore -v infra/secrets/age.agekey. Publishing the private key retroactively decrypts every ciphertext in git history.
  2. Plaintext Secret manifests. scripts/check-infra.sh fails if any file under infra/**/secrets/ lacks SOPS metadata or carries a data/stringData value that is not ENC[...] ciphertext, and the repository gitleaks/trivy secret scans stay green because SOPS ciphertext contains no recognizable credential.
  3. Real values as "temporary" placeholders. The only committed plaintext key-like value in the platform is the agentgateway marker, which the gateway enforces as a demo API key (Chapter 5.5); anything with actual authority goes through the encrypted path or, better, never exists thanks to Workload Identity.

If a private key does leak, treat every secret encrypted to it as compromised: rotate the credentials themselves, not just the key — git history preserves the old ciphertext forever.

What is the gateway checkpoint?

With the local profile, call all three gateway ports through one port-forward and inspect :15020/metrics. Verify no service has type LoadBalancer or NodePort:

kubectl -n agentops get svc -o custom-columns=NAME:.metadata.name,TYPE:.spec.type